Informing multisource decoding in robust automatic speech recognition

Listeners are remarkably adept at recognising speech in natural multisource environments, while most Automatic Speech Recognition (ASR) technology fails in these conditions. It has been proposed that this human ability is governed by Auditory Scene Analysis (ASA) processes, in which a sound mixture is segregated into perceptual packages, called ‘streams’, by a combination of bottom-up and top-down processing. This thesis examines a novel ASR framework based on the ASA account, Speech Fragment Decoding (SFD). A ‘fragment’ is a spectro-temporal region where energy from a single sound source dominates. SFD employs techniques developed from knowledge about the auditory system to identify fragments. A decoding process using statistical speech models is applied to the fragment representation to simultaneously identify speech evidence and recognise speech. In this study three techniques for improving SFD are investigated. Firstly, explicit duration modelling is exploited to combat the corruption of acoustic data which often causes the decoder to produce word matches with unrealistic durations. Secondly, it is argued that the top-down information in recognition models may be insufficient to mediate the speech identification. Knowledge that can assist the decoder in the choice of speech evidence is investigated. Thirdly, pitch cues derived from structure in the correlogram are used in the fragment generation process. A range of small-vocabulary speech recognition experiments are conducted for evaluation. The improved SFD system is able to produce word error rates significantly lower than conventional ASR, and is relatively insensitive to a range of noise conditions. In conclusion, the framework provides some progress towards finding a general solution to the robust ASR problem in multisource environments.